Evaluation and Policy Analysis

The D.A.R.E. Program

What do we know about the D.A.R.E. Program?

Program Evaluation Results

RTI’s Evaluation Results

(images: wikipedia)

A Brief History of Evaluation Research

(images: City of Columbus)

Joint Committee’s Standards of Evaluation

4 Features all evaluations should have:

Utility - ensures that an evaluation will serve the practical information needs of intended users.
Feasibility - ensures that all evaluation will be realistic, prudent, diplomatic, and frugality
Propriety - ensures that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation
Accuracy - ensures that an evaluation will reveal and convey technically adequate information about the features that determine worth of the program

Evaluation Basics

Evaluation Alternatives

All evaluation is empirical and data driven. Objective and empitical assemnets of policies and programs are the corner stone of the evaluation field.

Evaluation of need (Is the program needed?)

Evaluability assessment (Can the program be evaluated?)

Evaluation process (How does the program operate?)

Evaluation of impact (What is the program’s impact?)

Evaluation of efficiency (How efficient is the program?)

Evaluation Alternatives

Impact Evaluation - Analysis of the extent to which a treatment or another service has an effect

Efficiency Analysis - a type of analysis that compares program cost, with program effects.

1.) Cost-benefit analysis - Compares program cost with the economic value of the program 


2.) Cost effectivness analysis - comapres cost with actual program outcomes.

Case Study

Cost-Benefit Analysis of Therapeutic Communities

Sacks et al., (2002) conducted a cost-benefit analysis of modified therapeutic communities (TC).

(images: ENSO Recovery)

A therpeuatic community is an alternation to the traditional correctional response to drug addiction, which typically involves incarceration for those possessing or trafficking of illegal substances.

Within the communities, abusers participate in an intensive, structure living experience with other addicts who are attempting to stay sober.

Although these program are expensive.

Design

They assessed 342 homeless, mentally ill chemical absuers who were randomly assigned into the treatment center group or the treatment as usual group.

The Dvs were employment status, criminal activity, and utilization of health services.

Results

The average cost of TC treatment was 20,361, compared to the traditional economic cost of normal TC which was 305,273.

The cost benefit ration was 13:1. Suggesting the TC program had beneficially results for its cost.

Design Decisions

Black Box Evaluations or Program Theory

Black Box Evaluations

The focus is on whether cases seem to have changed as a result of the exposure to the program.

(images: SlidePlayer)

Design Decisions

Black Box Evaluations or Program Theory

Program Theory

Describes what has been learned about how the program is effective.

(images: giphy)

Design Decisions

Researcher or Stakehold Orientation

Stakeholder Approach

Encourages researchers to be responsive to program stakeholders.
Issues for study are centered on the views of people involved with the program, and reports are made for the participants.
Utilization-focused evaluation the evaluator forms a task force that help to shape the evaluation project so they are more likely to get successful results.

Design Decisions

Researcher or Stakehold Orientation

Social Science Approach

Emphasizes the importance of researcher expertise and maintenance of some autonmy to develop the most trustworthy unbiased program evaluation.

Integrated Approaches

Attempt to cover issues of conern to both stakeholders and evaluators.

Evaluation in Action

Case-Study

Problem-Oriented Policing in Violent Crime Areas

Braga et al., (1999)

Randomized experimental design study used to evaluate Problem-Oriented Policing

Studies have found that over half of all crimes in a city are committed within a few criminogenic places. These places are termed hot spots.
The strategies are to reduce crim in these high-activity crime places. It challenges officers to identify and analyze the causes of problems behind a string of criminal incidents. Once the problem is known, police can implement a response and thus reduce crime overall.

Braga et al., (1999)

Study Design

Use the diagram on p. 239 to describe the experiment design from Braga on the board.

examined the efficacy of the POP strategies, by using three separate DVs - incident report data, 911 calls, and physical observation (e.g., physical disorder like trash on the street, drinking in public, graffiti.
if the POP strategies were effect, they should be a decrease in these DVs

Results:

The POP strategies examined in the evaluation, were shown to be successful at reducing the DVs.

Strength of Randomized Experimental Designs in Impact Evaluations

Braga et al., (1999) used a true experimental design.

What are the three elements of experimental design?

Are these finding generalizable? In other words, can we apply them to a larger population?

Qualitative and Quantitative Methods

Traditionally, evaluation methods are quantitative.

Although, qualitative methods can often offer more depth in understanding the program effectiveness. For example, figuring out what inside ‘the black box.’ This can be completed with intense interviewing of staff or clients.

Usually, the more complex the program, the more effective qualitative methods are.

Increasing Demand for Evidence-Based Policy

Increasing Demand for Evidence-Based Policy

The Campbell Collaboration

An international research network
Purpose is to prepare and disseminate systamic reviews of social science evidence in three fields - criminal justice, education, and social welfare.
For more information click here

Ethics

These programs directly affect people’s lives. Therefore we must be honest and transparent with how the studies are conducted.

Furthermore, letting programs continue that have no evidence of working (e.g., the D.A.R.E. program) takes money away from other programs that may benefit people’s lives.

This is why policy and evaluation research is so important.

Thanks for coming! Have a good weekend!

(images: giphy)